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COMPARATIVE  ACCURACY  OF  FIVE  INDICES  OF  DIMENSIONALITY 

OF  BINARY  ITEMS 


Introduction 

The  standard  method  of  testing  the  dimensionality  of  binary  items  is 
to  comoute  tetrachoric  intercorrelations,  obtain  the  principal  components, 
and  inspect  the  latent  roots.  If  the  items  are  unidimensional,  one  expects 
the  first  root  to  describe  a  major  although  numerically  uncertain  propor¬ 
tion  of  the  total  variance.  One  also  expects  a  large  difference  in  the 
size  of  the  second  root  relative  to  the  first  and  no  sizable  gap  subse¬ 
quent  to  the  first  difference.  These  evaluations,  however,  have  no  firm 
quantitative  basis. 

Tetrachoric  correlations  are  used  for  the  component  analysis  because 
the  size  of  product-moment  correlations  is  affected  by  the  difficulties  of 
the  items.  Everyone  knows  that  these  correlations  (phi  coefficients)  pro¬ 
duce  difficulty  level  factors.  If  items  were  to  produce  a  perfect  Guttman 
scale,  the  matrix  of  intercorrelations  would  be  a  perfect  simplex,  not  a 
Spearman  hierarchy. 

Even  though  the  use  of  tetrachorics  is  more  or  less  standard,  the  R- 
matrix  has  two  undesirable  properties  for  purposes  of  factor  analysis  and 

the  determination  of  dimensionality.  The  matrix  is  very  likely  to  be  non- 

»  • 

Gramian,  and  the  samplinq  errors  of  the  individual  tetrachorics  vary  widely 
as  a  function  of  the  difficulty  levels  of  the  items  correlated.  The  effects 
of  these  properties  become  more  severe  as  the  spread  of  item  difficulties 
increases. 

Importance  of  Dimensionality.  The  problem  of  dimensionality  has  come 
to  the  fore  with  the  increase  o*  interest  in  Item  ResDonse  Theory  and  the 
use  of  that  methodology  in  adaptive  testing.  Although  dependable  item  and 
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person  parameters  can  be  obtained  from  item  pools  that  are  not  unidimen- 
sional  in  the  strict  sense  of  that  term— a  dominant  dimension  defined  by 
correlated  group  factors  is  sufficient  (Drasgow  and  Parsons,  1983)  and  more 
valid  in  most  applications  (Humphreys,  1 985 ) — a  method  of  test  administra¬ 
tion  that  potentially  provides  a  different  set  of  items  for  each  examinee 
and  regularly  provides  a  different  set  of  items  for  subgroups  of  examinees 
places  a  greater  demand  on  the  IRT  assumption  of  uni  dimensional ity. 

It  becomes  important,  therefore,  to  determine  the  accuracy  with  which 
a  decision  contrasting  unidimensionality  with  multidimensionality  can  be 
made  from  the  matrix  of  tetrachoric  correlations.  It  is  also  important  to 
evaluate  nontraditional  ways  of  evaluating  dimensionality.  In  order  to  do 
this  it  is  essential  to  have  a  model  of  known  dimensionality,  to  sample 
both  items  and  examinees,  and  to  compare  possible  indices  of  dimensionality 
in  the  same  data  sets. 

Characteristics  of  a  Desirable  Model 

Our  objective  was  to  develop  a  model  that  could  simulate  realistically 
psychological  data,  especially  cognitive  data.  The  starting  point  was  the 
model  developed  by  Tucker,  Koopman,  and  Linn  (1969).  Those  authors  discussed 
both  a  "formal"  factor  model  and  a  "simulation"  model.  The  models  differ 
in  that  the  latter  adds  a  relatively  small  amount  of  variance  from  a  large 
number  of  small,  overlapping  factors.  The  added  variance  simulates  the 
large  number  of  determinants  of  responses  to  test  items.  Neither  items  nor 
tests  are  ever  pure  measures  of  a  hypothetical  factor.  We  plan  to  investigate 
both  models  as  they  affect  detection  of  multidimensionality,  but  we  shall 
initially  report  results  for  the  formal  model  only. 

A  second  requirement  was  that  the  model  should  furnish  data  realistic 
in  its  psychometric  properties.  Distributions  of  item  difficulties  should 
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approximate  those  that  are  encountered  in  practice.  Item  correlations 
should  be  at  levels  encountered  in  most  item  pools.  Product-moment  corre¬ 
lations  among  cognitive  items  in  wide  ranges  of  talent  are  rarely  greater 
than  .25  and  are  frequently  considerably  lower.  Differences  in  the  size  of 
mean  correlations  between  a  rather  heterogeneous  intelligence  test  and  a 
homogeneous  test  of  mechanical  information  are  small.  Tetrachoric  corre¬ 
lations  are  larger,  but  generally  well  below  the  maximum  value  of  1.00 
expected  in  a  perfect  Guttman  scale.  Tetrachoric  correlations  of  unity  do 
occur  in  samples  of  reasonable  size  but  only  when  a  very  easy  item  is 
correlated  with  a  very  difficult  item.  A  zero  frequency  in  one  cell  of  the 
four-fold  table  can  readily  occur  by  chance  in  such  combinations. 

A  third  requirement  is  that  the  model  should  be  a  compensatory  one. 

In  this  initial  study,  an  item's  variance  is  broken  down  into  a 

linear  combination  of  loadings  on  major  group  factors  and 

random  error.  Intercorrelations  are  the  sums  of  products  of  overlapping 

factors.  A  compensatory  model  seems  to  fit  Dsychological  data  rather  well, 

including  the  predictive  validities  of  psychological  tests  for  practical 

criteria. 

Description  of  the  Model 

The  data  in  this  study  were  generated  in  such  a  way  that  the  struc¬ 
tural  relations  desired  were  known  and  prespecified.  This  was  desired  so 
that  the  validity  of  the  factor  analytic  design  could  be  checked.  With 
such  a  generation  method,  a  design  plan  can  be  adjusted  by  the  experi¬ 
menter  in  terms  of  the  exact  nature  and  quality  of  the  data. 

Basic  Notions.  The  model  used  here  specifies  the  existence  of  a  major 
domain,  which  represents  the  most  important  influences  on  individuals'  observed 
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scores.  Consider  factor  matrix  B  whose  entries  are  the  actual  input  loadings. 
Every  row  (i)  in  B  represents  a  variable,  while  every  column  (j)  in  B 


represents  a  factor.  Each  element  in  B  (B^)  is  an  input  factor  loading. 

These  elements  are  the  conceptual  loadings  or  values  in  the  major  domain 
that  represent  ideas  or  theoretical  notions  concerning  the  makeup  and 
structural  relations  of  the  input  variables  under  study. 

The  major  domain  was  presumed  to  have  simple  structure.  Further,  a 
restriction  of  the  factor  matrix  to  positive  manifold  was  desired.  This 
latter  characteristic  was  required  because  the  data  being  simulated  were 
cognitive  ability  items. 

Factor  Intercorrelations.  An  initial  step  in  the  process  of  generating 
the  factor  model  was  determining  the  phi  matrix  (0,  intercorrelations  among 
factors).  Significant  control  of  this  parameter  was  available  to  the  experimenter. 

A  value  is  set  by  the  experimenter  in  the  range  0,1  that  will  indirectly 
determine  a  minimum  level  of  the  factor  correlations.  A  second,  higher 
value  in  the  same  range  is  then  chosen  from  which  the  first  is  subtracted. 

The  difference  is  multiplied  by  a  random  number  in  the  interval  0,1  and  the 
product  added  t.o  the  initial  value.  The  procedure  is  repeated  for  each  factor 
in  the  model  using,  Tor  the  present,  the  same  first  and  second  values.  The 
result  is  a  unique  value  for  each  factor. 

The  next  step  determines  the  correlation  between  pairs  of  factors. 

This  involves  taking  the  square  root  of  the  values  determined  for  each 
factor  in  the  preceding  step  and  obtaining  the  cross-products  among  all 
factors.  In  effect,  the  factor  intercorrelations  are  the  products  of  factor 
loadings  on  a  single  general  factor. 
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The  factor  correlations  used  in  the  present  application  of  the  model  were 
determined  by  initial  values  of  .20  and  .40  for  the  low  and  .50  and  .70  for  the 
high  levels,  respectively.  Thus  the  random  increments  to  the  base  were,  on 
average,  the  same  size  for  the  two  levels  of  factor  correlations. 

Ensuring  Simple  Structure.  The  next  step  in  the  generation  of  the 
major  domain  is  deriving  the  first  order  B  having  simple  structure  with  a 
restriction  to  positive  manifold.  Values  for  the  intial  B  are  chosen  by 
selecting  a  random  value  in  the  interval  0,1. 

Several  checks  are  carried  out  at  this  point  to  assure  that  the 
structural  relations  and  the  general  quality  of  the  factor  matrix  are 
known.  These  are  listed  here: 

-~sj 

1.  A  cut  off  probability  for  establishing  zeros  in  B  is  used  at  this 
point  to  determine  the  actual  number  of  zeros  in  each  column  of  B  (i.e.,  per 
factor).  In  effect,  this  step  establishes  the  simple  structure  aspect  of 
the  factor  matrix. 

2.  A  second  precaution  is  carried  out  to  insure  the  presence  of  at 
least  one  hiah  (  nonzero  )  entry  in  each  row  (i.e.,  per  variable)  of  the  factor 
matrix.  If  no  nonzero  loadinq  is  present,  one  is  inserted  in  one  of  the 
columns.  The  position  of  the  nonzero  entry  is  chosen  randomly. 

3.  The  third  check  concerns  a  count  of  the  high  loadings  in  each  column 
of  the  factor  matrix  (i.e.,  for  each  factor).  Again,  the  investigator 
determines  a  priori  the  minimum  number  of  high  loadings  desired.  As  before, 
hiqh  entries  are  inserted  until  the  minimum  number  of  high  loadings 
desired  is  achieved. 

4.  Lastly,  all  pairs  of  columns  (,i  and  k,  j  f  k)  are  compared  to 
determine  whether  there  are  the  appropriate  (specified  a  priori )  number  of 
rows  with  a  high  entry  on  i  and  zero  entry  on  k,  aoain  a  further  assurance 


that  each  variable  is  represented  and  that  simple  structure  is  achieved. 
Adjustments  are  made  when  necessary. 

Size  of  Factor  Loadings.  A  final  set  of  adjustments  is  carried  out  to 
effect  the  actual  size  of  the  loadinqs  in  the  factor  matrix  (loadings  are 
either  0.0  or  .5  before  this  adjustment).  The  investigator  again  has  input 
into  the  absolute  size  of  factor  loadings.  The  investigator  determines  two 
initial  values  which  indicate  the  range  from  which  loadings  for  the  nonzero 
entries  in  the  factor  matrix  are  chosen.  This  process  is  the  same  as  that 
used  for  the  generation  of  the  0  matrix.  The  procedure  presented  earlier  for 
second  order  G1  coefficients  is  used  at  this  point  also.  A  vector  of  such 
coefficients  is  obtained,  one  coefficient  for  each  variable.  These 
coefficients  are  random  rectilinearly  distributed  from  Gl$  to  Gl^.  (Gl$  and 
Gl^  are  defined  as  above  and  are  selected  a  priori  by  the  investigator). 

The  G1  coefficients  are  the  hypothesized  variance  from  the  major  conceptualized 
domain  of  influences. 

At  this  point,  the  matrix  A,  which  is  the  factor  weight  matrix 
transformed  to  uncorrelated  factors,  is  obtained.  The  first  step  in  this 
process  is  computing  the  Cholesky  decomposition  (vj  of  the  matrix  phi  (0) 
determined  as  above.  Matrix  B  is  post  multiplied  by  v  to  obtain  an 
initial  A  matrix.  Next,  rows  of  A  and  R  are  adjusted  to  take  into  account 
the  vector  of  computed  G1  coefficients  (again,  the  desired  variance 
attributable  to  the  major  domain  of  influences).  After  rows  of  A  are 
normalized,  the  same  multiplier  is  applied  to  B  to  rescale  B  in  accordance 
with  the  transformation  to  A. 

Application  to  Binary  Scores.  Up  to  this  point  the  model  has  been 
described  in  terms  of  continuous  variates.  The  model  in  this  section  is 
converted  to  binary  measures.  (Each  individual  has  a  measure  (score)  on  each 


of  the  k  dimensions.)  Definitions  for  scores  on  dimensions  and  their  components 
are  as  follows: 


=  score  or  measure  of  individual  i  on  dimension  k. 
=  item  score  of  individual  i  on  item  j. 

=  weight  for  item  j  on  dimension  k. 

A  fundamental  relation  is: 


o)  iir 


k  xikwjk 


To  generalize  this  relation,  we  can  see  that: 


(2)  Zi  =  X_-Q 1 

where:  X_^  is  a  vector  of  scores  X^,  and  Z^j  is  a  vector  of 
scores  Z^.  a  =  matrix  of  weights  W^. 

A  further  look  at  the  n  (weight)  matrix  reveals  "common"  factor  weights, 
"specific"  factor  weights,  and  "error  of  measurement"  factor  weights.  Specific 
and  error  factors  are  also  considered  together  as  uniqueness  component  or  U. 

We  designate  the  common  factor  weight  matrix  A  as  before. 

The  weight  matrix  a  can  be  decomposed  from  a  super  matrix  to  two  separate 
weiqht  matrices,  A  and  U  representing  weights  from  common  influences  and  unique 
influences,  respectively.  For  the  purposes  of  this  study,  the  vector  X^  can 
be  similarly  partitioned  into  common  and  unique  components  a^- ,  u^ ,  respectively, 

X  .  =(a.,  u.>.  Expanding  the  above  relation  (2),  we  see: 

111  • 

(3)  Z.  .  =  a  .A  +  u  •  U 

ij  -i  -J 

With  this  basic  factor  model  as  a  start,  individual  binary  data  (scores)  are 
derived  that  conform  to  the  qualities  of  the  simulation  model  in  continuous  form. 
The  factor  model  is  a  linear  model  involving  the  addition  of  the  contributions 
to  item  scores.  The  factor  weight  matrix  A  (common  factor  influences)  is  applied 
to  the  underlyina  score  distribution  and  combined  with  the  unique  components 
to  determine  item  scores.  To  compute  Z- ^  (item  score  of  individual  i  on  item  j) 
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from  (1),  a  random  normal  deviate  for  individual  i  on  each  dimension  k  (Xik) 
is  chosen  and  multiplied  by  factor  loading  from  A  for  item  j  on  dimension  k  (a^). 

For  a  three-factor  model,  the  following  holds: 

(4)  zij=Xiiaji  +  Xi2aj2  +  X  i 3aj3 
To  the  score  Z,  .  is  added  the  uniqueness  component.  Again,  a  random 

normal  deviate  is  chosen  (one  for  each  item)  which  is  multiplied  by  the  uniqueness 

for  that  item.  This  product  is  added  to  the  from  the  common  factors. 

Generation  of  Binary  Responses.  To  qenerate  item  binary  scores,  an  individual's 

normally  distributed  continuous  score  for  item  j  (2.—)  is  compared  to  a  cutting 

score  or  threshold  for  item  j.  The  cuttinq  score  distribution  is  derived  with 

input  from  the  investigator.  The  cutting  score  mean  and  standard  deviation  are 

determined  a  priori  and  are  used  to  set  up  the  cutting  score  values  (d - ) ,  one 

<J 

for  each  item  j.  Further,  a  vector  of  guessing  parameters  (c.)  is  set  up  taking 

J 

into  account  a  range  of  guessing  probabilities  specified  by  the  investigator. 

The  equations  that  characterize  the  probabilistic  model  used  for  generating  binary 
responses  follow.  S^-  represent  the  individual's  binary  response  to  item  j. 

S. .  is  initially  set  to  0. 

*  J 

5a)  IF  fZ„  >  dj)  Sij-1, 

5b)  IFdy.d,)  PKij-D-Cj. 

The  event  of  S^.  equalling  1  or  0  is  independent  from  occasion  to  occasion. 

Thus,  if  the  score  of  individual  i  to  item  j  is  greater  than,  or  equal  to,  the 
cuttinq  score  for  item  j,  the  individual  has  correctly  answered  the  item.  If  the 
score  (Z-jj)  is  less  than  the  threshold,  the  individual  receives  a  zero  to  indicate 
an  incorrect  response.  However,  a  guessing  component  is  taken  into  account  such  that 
the  individual's  wrong  answer  to  item  j  is  changed  to  a  correct  answer  with 
probability  Cj.  This  process  is  continued  for  all  j  items  of  the  i  individuals.  The 
model  can  be  seen  as  a  probabilistic  model  that  involves  eacn  measure  (item)  as  a 
sampling  of  influences  on  the  individual's  behavior. 
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Independent  Parameters 

Five  parameters  were  varied  in  a  factorial  design.  The  variations 
defined  a  range  of  possible  values  for  each  parameter,  but  no  attempt  was 
made  at  this  staqe  to  fill  in  the  gaps.  Cells  in  the  design  were  filled 
with  20  independent  samples  with  the  exception  of  those  defined  by  the 
largest  number  of  items.  Only  10  replications  were  obtained  in  the  latter 
case  to  conserve  computer  time. 

Sample  Sizes.  These  were  set  at  125  and  500,  providing  a  2  to  1  ratio 
of  expected  sampling  errors.  Because  this  parameter  is  under  the  control  of 
the  research  person,  an  adequate  sample  size  (JN)  can  usually  be  established 
in  advance.  All  indices  of  dimensionality  should  vary  in  accuracy  as  a 
function  of  this  parameter. 

Number  of  Items.  This  parameter  (n)  was  varied  as  follows:  20,  30,  40, 
and  60.  Number  of  items  can  also  be  directly  controlled  by  the  research 
person.  For  a  given  number  of  factors  one  expects  to  be  able  to  define 
factors  more  accurately  the  larqer  the  number  of  items. 

Item  Difficulties.  Two  distributions  were  used,  one  relatively  flat, 
the  other  peaked.  These  were  defined  in  normal  deviate  units  as  follows: 
mu  =  .10,  n.-=  .80;  mu  =  -.13,  o=  .32.  The  positive  sign  of  the  mean 
indicates  a  central  tendency  of  p-values  less  than  .50  in  the  absence  of 
guessing.  The  mean  p-values  for,  both  are  greater  than  .50  after  the 
quessinq  parameter  has  been  applied.  An  experienced  test  constructor  can 
exercise  an  approximate  degree  of  control  over  this  parameter.  More  import¬ 
antly,  item  difficulties  are  always  knowable  before  a  decision  concerning 
dimensionality  has  to  be  made.  Note  that  variation  in  the  distribution  of 
item  difficulties  affects  the  size  of  the  item  product-moment  intercorrela¬ 
tions.  It  affects  only  the  sampling  variability  of  tetrachorics. 


Factor  Intercorrelations.  The  manipulation  of  this  parameter  produced 
mean  correlations  of  approximately  .35  and  .55.  This  difference  in  size  did, 
of  course,  produce  variation  in  size  of  item  intercorrelations  of  both  types. 

For  purposes  of  determining  dimensionality  factor  intercorrelations  are  not 
knowable,  but  one  can  obtain  the  mean  item  correlation  in  advance  of  a  decision. 
For  a  given  set  of  items  the  mean  product-moment  correlation  is  readily  computed 
from  an  internal  consistency  statistic  for  the  total  score. 

Number  of  Factors.  This  parameter  was  varied  from  1  through  5.  A  test 
constructor  can  only  control  it  approximately  by  adherence  to  item  specifications 
However,  good  test  construction  practices  will  not  increase  the  number  of  major 
factors  as  the  number  of  items  is  increased.  On  the  other  hand,  the  size  of 
the  correlations  among  the  multiple  factors  that  remain,  when  unidimensionality 
is  not  achieved  by  the  item  writer  who  is  working  toward  that  goal,  is  likely 
to  be  high.  If  unidimensionality  can  be  rejected,  it  is  also  important  to 
decide  on  the  number  of  group  factors,  but  this  decision  is  not  central  to 
the  present  research.  It  should  also  be  noted  that  variation  of  this  parameter 
in  our  model  results  in  some  degree  of  variation  of  the  level  of  item  inter¬ 
correlations.  In  this  case  there  is  variation  in  both  product-moment  and 
tetrachoric  correlations. 

Given  the  oblique  factors  that  are  characteristic  of  cognitive  data,  one 
expects  increasing  difficulty  in  distinguishing  between  unidimensionality  and 
multidimensionality  as  the  number  of  factors  increases.  Positively  correlated 
first-order  factors  determine  one  or  more  second-order  factors,  but  in  our 
present  model  there  is  only  one  second-order  factor.  It  describes  an  increasing 
amount  of  total  variance  as  the  number  of  factors  increases.  When  the  general 
factor  is  held  constant,  the  total  contribution  of  five  factors  is  less  than 
the  total  contribution  of  two  factors  In  an  important  sense,  as  the  number 


n 

of  homogenously  correlated  group  factors  increases,  the  more  nearly  do  the 
data  appear  to  be  unidimensional  and  the  more  difficult  the  task  becomes 
in  both  our  model  and  real  data. 

Dependent  Variables 

In  addition  to  the  use  of  indices  based  on  nontraditional  approaches 
to  dimensionality  using  covariances  and  product-moment  correlations,  we 
also  used  indices  based  on  the  Eigenvalues  of  R-matrices  composed  of 
tetrachoric  correlations.  The  principal  factors  model,  substituting  the 
highest  correlation  in  a  column  for  the  unity  of  the  R-matrix,  was  used 
for  these  indices  in  place  of  the  principal  components.  This  decision 
was  based  on  preliminary  research  in  which  the  differences  between  the 
two  models  were  trivial  in  size,  but  on  average  favored  the  principal  factors. 

Eigenvalue  Differences.  The  size  of  the  difference  between  the  first 
two  Eigenvalues  may  represent  most  closely  the  basis  for  a  decision  concerning 
dimensionality  obtained  by  "root  starinq",  i.e.,  by  inspection  of  the  pattern 
of  the  latent  roots.  Because  the  size  of  the  roots  is  a  function  of  the 
level  of  item  intercorrelations,  the  absolute  size  of  the  difference  must 
be  standardized.  Each  difference,  therefore,  was  divided  by  the  mean  of 
the  first  of  the  two  Eigenvalues.  This  standardized  difference  approximates 
the  information  that  would  be  furnished  by  an  index  consisting  of  the  ratio 
of  the  first  two  Eigenvalues  for  each  sample  in  the  cell. 

Ratios  of  Differences.  A  ratio  of  the  intial  difference  to  later 
differences  better  represents  the  shape  of  the  curve  of  the  Eigenvalues. 

In  this  research  we  computed  the  ratio  of  the  difference  between  the  first 
two  roots  to  the  mean  of  the  following  two  differences.  If  a  set  of  items 
is  measuring  a  sinqle  dimension,  not  only  should  there  be  a  large  drop  in 
size  from  the  first  root  to  the  second,  but  subsequent  decreases  should  be 


Local  Independence  Criterion.  If  unidimensionality  is  present,  the 
intercorrelations  of  items  in  the  population  of  persons  who  are  at  the  same 
level  of  the  latent  trait  are  equal  to  zero.  Local  independence  can  be 
approximated  in  fallible  data  by  restricting  analyses  to  those  who  have  the 
same  total  score  on  the  set  of  items.  One  cannot  hope  to  find  an  index  of 
dimensionality  by  analyzing  ^different  matrices  independently,  one  for  each 
level  of  total  score,  so  the  following  procedure  was  developed: 

1)  An  aggregate  variance-covariance  matrix  is  formed  from  the  separate 
matrices  computed  in  samples  of  persons  having  the  same  total  score.  Each 
sub-matrix  is  weighted  by  its  sample  size  in  forminq  the  aggregate. 

2)  Signs  of  the  aggregate  covariances  are  changed  in  accordance  with  the 
sign-changing  procedure  of  centroid  factor  analysis  to  maximize  the  algebraic 
sum  of  the  C-matrix.  Because  this  C-niatrix  held  total  score  constant,  there 
are  approximately  equal  numbers  of  positive  and  negative  signs  in  the 
aggregate  matrix. 

3)  The  ratio  of  the  algebraic  sum  of  covariances  following  the  sign 
change  to  the  absolute  sum  is  formed.  Ratios  that  approach  unity  indicate 
the  presence  of  more  than  one  factor  among  the  original  item  covariances; 
i.e.,  there  is  structure  remaining  in  the  matrix  although  one  factor  has 
been  removed. 

4)  The  ratio  in  step  3  is  then  compared  with  the  ratio  of  algebraic 
to  absolute  sum  of  the  values  in  the  covariance  matrix  of  raw  scores  in 
which  total  score  is  not  held  constant.  For  this  ratio  the  sign  change  was 
placed  in  the  program  so  that  the  technique  would  be  applicable  to  matrices 
composed  of  both  positive  and  negative  correlations  among  noncognitive  items. 


5)  The  comparison  at  this  point  in  our  research  involves  subtracting  (3) 
from  (4).  Among  reliable  cognitive  items  the  ratio  in  (4)  will  closely 
approach  unity  whatever  the  dimensionality  may  be.  The  presence  of  multiple 
factors  is  indicated,  therefore,  by  small  differences. 

These  ratios  were  computed  with  and  without  the  diagonal  entry.  Results 
have  been  highly  similar,  but  ratios  from  which  diagonals  were  omitted  had  a 
slight  edge  in  accuracy.  Our  results  section  will  present  data  only  for  the 
latter  alternative. 

Pattern  in  Factor  Loadings.  The  first  and  second  principal  components 
of  the  product-moment  intercorrelations  of  items  comprising  a  perfect  Guttman 
scale  have  distinctive  patterns.  The  loadings  on  the  first  component  are  all 
positive  with  the  largest  loadings  being  associated  with  items  of  moderate 
difficulty  level  and  the  smallest  with  the  easiest  and  the  most  difficult 
items.  The  loadings  on  the  second  component  form  a  curve  that  approximates 
an  ogive  with  easy  and  difficult  items  having  high  loadings  of  opposite  sign 
and  with  items  of  moderate  difficulty  having  loadings  close  to  zero.  Because 
a  perfect  Guttman  scale  is  also  a  perfect  index  of  unidimensionality,  it  was 
our  hypothesis  that  patterns  of  loadinqs  in  the  first  two  principal  factors 
of  the  product-moment  R-matrix  could  be  used  as  an  index  of  dimensionality 
in  fallible  item  data.  A  critical  assumption  in  the  application  of  this 
hypothesis  is  that  there  is  orthogonality  of  item  difficulties  and  factor 
content  when  more  than  one  factor  is  present  in  the  items. 

One  index  made  use  of  the  signs  of  the  second  principal  factor  of  the 
R-matrix  of  product-moment  correlations. 

1)  The  matrix  is  factored  after  replacing  the  unities  in  the  diagonal 
with  squared  multiple  correlations. 

2)  Items  ere  ranked  in  order  of  difficulty  and  the  preponderance  of  signs 


of  the  loadings  on  the  second  principal  factor  in  the  easy  and  difficult 
halves  of  the  items  determined. 

3)  Each  aberrant  item,  e.g.,  one  with  a  negative  sign  in  the  easy  half 
when  most  items  in  that  half  have  positive  signs,  is  given  a  numerical  value 
based  on  the  number  of  ranks  by  which  it  is  removed  from  the  center  of  the 
distribution  of  difficulties. 

4)  These  numerical  values  are  summed  over  all  aberrant  items.  Small 
sums,  in  accordance  with  our  hypothesis,  should  be  associated  with  unidimen¬ 
sionality. 

A  second  index  that  is  sufficiently  independent  of  the  first  to  be  given 
separate  consideration  is  based  on  information  obtained  from  both  the  first 
and  second  principal  factors  among  the  product-moment  correlations. 

Steps  (1)  and  (2)  are  the  same  as  for  the  first  index. 

3)  Each  aberrant  item  is  given  a  numerical  value  representina  the  product 
of  its  first  and  second  factor  loadings. 

4)  These  numerical  values  are  summed  without  regard  to  sign  over  all 
aberrant  items.  Small  sums  are  again  associated  with  unidimensionality. 

The  rationale  for  this  index  is  that  items  so  unreliable  as  to  have  little 
in  common  with  other  items  in  the  test  should  not  influence  the  decision 
concerning  dimensionality.  On  the  other  hand,  items  that  do  have  something 
in  common  with  at  least  a  subset  of  other  items  and  have  large,  aberrant 
second  factor  loadings  should  strongly  influence  the  dimensionality  decision. 

Results 

Levels  of  Intercorrelations.  Before  presenting  the  results  from  the 
various  indices  of  dimensionality,  it  will  be  useful  to  have  as  background 
the  levels  of  intercorrelations  (phis)  produced  by  the  three  parameters  for 
which  effects  were  expected.  Typical  product-moment  correlations  in  the 
samples  appear  in  Table  1. 
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Table  1 

Representative  Mean  Item  Correlations 
As  a  Function  of  Three  Parameters 

Number  of  Factors 


1 

2 

3 

4 

5 

High  Low 

High  Low 

High 

Low 

High  Low 

Wide  .215 

.18  .17 

.16  .14 

.16 

.135 

.16  .13 

Narrow  .30 

.24  .22 

.23  .20 

.225 

.19 

.225  .18 

Table  2 

shows  the  Kuder-Richardson  coefficients  for  several 

selected 

levels  of  item  intercorrelations 

that  cover 

the 

range  i 

Df  values 

in  Table  1. 

Table  2 

Kuder- Richards on  Coefficients  As 

a  Function 

of 

Number 

of  Items  and 

Selected  Values  of  Mean  Item  Correlations 

Mean  Item 

Number  of 

Items 

Correlations 

20 

30 

40 

60 

.30 

.90 

.93 

.94 

.96 

.25 

.87 

.91 

.93 

.95 

.20 

.83 

.88 

.91 

.94 

.15 

.78 

.84 

.88 

.91 

.12 

.73 

.80 

.84 

.89 

BSU  6  XJ 


Although  we  tried  to  maintain  a  fairly  constant  level  of  item  correla¬ 
tions  for  factors  2  through  5  by  manipulating  the  proportion  of  items  loaded 
on  a  sinqle  factor,  we  were  not  quite  successful.  The  larqe  gap,  however, 
is  between  one  factor  and  multiple  factors  because  there  were  no  zero 
loadings  in  the  one  factor  model.  The  distribution  of  item  difficulties  has 
a  large  effect  as  required  by  the  nature  of  the  correlations.  The  typical 
item  correlation  changes  least  as  a  function  of  the  amount  by  which  we 
manipulated  the  intercorrelations  of  the  factors  in  the  continuous  bivariate 
model.  Even  so,  as  will  be  seen  later,  the  differences  in  size  of  the  factor 
intercorrelations  have  important  consequences  for  the  accuracy  of  the  indices. 

The  range  of  Kuder-Richardson  coefficients  demonstrates  that  our  item 
correlations  are  in  the  appropriate  ball  park  for  most  cognitive  tests. 

Varying  the  number  of  items  compensates  for  low  levels  of  item  correlations. 
Perhaps  the  value  that  is  most  out  of  line  is  for  one  factor  in  the  narrow 
distribution  of  item  difficulties.  Mean  item  correlations  of  .30  in  cognitive 
tests  are  rare. 

A  Measure  of  Overlap.  The  accuracy  with  which  one  can  reject  a  partic¬ 
ular  null  hypothesis  depends  on  the  overlap  of  two  sampling  distributions. 

In  the  present  research  we  obtained  sampling  distributions  of  indices  in  item 
intercorrelations  that  were  based  on  major  factors  varying  from  one  through 
five.  Because  the  distributions  of  several  indices  were  skewed,  we  realized 
belatedly  that  means  and  standard  deviations  of  these  distributions  were 
inadequate  for  determining  overlap.  In  this  initial  report  we  obtained  our 
measure  from  hand-tallied  distributions.  The  procedure  is  as  follows: 

In  the  distribution  of  a  given  index  for  one  factor  we  select  the 
value  that  represents  one  factor  least  well.  If  low  values  of  the  index  are 
expected,  this  is  the  highest  value.  We  then  count  the  number  of  values  in 


each  of  the  distributions  of  multiple  factors  that  were  equal  to  or  lower 
than  our  critical  first  factor  value.  Next  we  select  in  the  distributions 
of  each  of  the  multiple  factors  the  value  that  represents  the  presence  of 
more  than  one  factor  least  well.  Again,  if  high  values  for  multiple  factors 
are  expected,  this  is  the  lowest  value.  We  count  the  number  of  one-factor 
values  equal  to  or  hiqher  than  the  multiple  factor  critical  value.  Skewness 
results  in  asymmetry  of  the  two  measures  of  overlap,  so  we  add  them  together 
to  obtain  a  single  value  that  can  vary  from  zero  to  40  (20  for  60  items). 

Arrangement  of  the  Tables.  Tables  are  arranged  in  accordance  with  a 
standard  pattern.  Small  sample  size  is  in  the  upper  half,  large  in  the  lower 
half.  Columns  are  defined  in  the  first  instance  by  the  number  of  items  in 
the  test  and  secondarily  by  the  two  levels  of  factor  intercorrelations.  One 
factor  data,  of  course,  do  not  vary  along  this  parameter.  Rows  are  defined 
in  the  first  instance  by  the  comparisons  of  one  factor  in  turn  with  two,  three, 
four,  and  five.  A  second  distinction  within  each  of  these  comparisons 
involves  the  two  levels  of  item  difficulties. 

Standardized  Difference  in  Eigenvalues.  Overlap  information  for  this 
index  is  presented  in  Table  3.  It  is  clear  that  this  index  represents  a 
highly  unreliable  method  of  drawing  inferences  concerning  dimensionality. 

In  only  a  small  number  of  the  most  favorable  combinations  of  parameters  did 
we  obtain  zero  overlap  in  20  replications.  Although  the  results  would  not 
be  identical,  the  ratio  of  the  first  two  Eigenvalues,  for  which  the  present 
index  is  an  approximation,  can  also  be  considered  to  have  little  promise. 

Ratio  of  Differences.  This  index,  for  which  overlap  information  is 
summarized  in  Table  4,  is  moderately  effective  for  the  restricted  distribution 
of  item  difficulties,  but  quite  ineffective  in  the  wide  range.  In  contrast 
to  the  standardized  differences  in  Eigenvalues,  ratios  of  differences  reflect 
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Table  3 

Amount  of  Overlap  in  Each  Cell 
Based  on  the  First  Difference  in  Roots 

N=1 25 

20  30  40  60* 


High 

Low 

High 

Low 

High 

Low 

High 

Low 

1  vs.  2 

Wide 

39 

38 

38 

33 

35 

31 

30 

22 

Narrow 

33 

27 

33 

29 

31 

28 

16 

22 

1  vs.  3 

Wide 

38 

38 

38 

29 

34 

31 

38 

22 

Narrow 

32 

25 

35 

27 

33 

20 

22 

20 

1  vs.  4 

Wi  de 

37 

37 

35 

31 

36 

28 

38 

30 

Narrow 

38 

29 

34 

31 

30 

20 

26 

20 

1  vs.  5 

Wide 

37 

38 

39 

36 

37 

32 

34 

30 

Narrow 

31 

30 

35 

31 

32 

27 

30 

26 

N=500 

20 

30 

40 

60* 

High 

Low 

High 

Low 

High 

Low 

High 

Low 

1  vs.  2 

Wide 

39 

37 

31 

12 

33 

27 

32 

0 

Narrow 

20 

9 

17 

1 

22 

0 

6 

0 

1  vs.  3 

Wide 

38 

39 

34 

29 

34 

28 

34 

18 

Narrow 

24 

14 

26 

9 

26 

8 

12 

8 

1  vs.  4 

Wide 

40 

40 

35 

30 

36 

32 

32 

23 

Narrow 

29 

20 

28 

2 

33 

14 

20 

12 

1  vs.  5 

Wide 

40 

36 

33 

33 

39 

36 

32 

32 

Narrow 

28 

25 

30 

17 

31 

13 

24 

8 

♦Error  frequencies  doubled. 


wry 


Amount  of  Overlap  on  Each  Cell 

Based  on  the  Ratio  of  Successive  Differences 

N=125 


20 

30 

40 

60* 

High 

Low 

High 

Low 

High 

Low 

High 

Low 

1  vs.  2 

Wide 

36 

32 

35 

25 

38 

31 

0 

0 

Narrow 

3 

0 

6 

6 

0 

0 

0 

0 

1  vs.  3 

Wide 

36 

39 

35 

25 

37 

36 

20 

0 

Narrow 

14 

0 

6 

0 

0 

0 

0 

0 

1  vs.  4 

Wide 

39 

35 

39 

27 

40 

32 

38 

28 

Na  rrow 

18 

6 

8 

6 

4 

0 

0 

0 

1  vs.  5 

Wide 

40 

33 

39 

35 

40 

34 

34 

36 

Narrow 

16 

2 

22 

6 

9 

2 

0 

0 

N=500 

20 

30 

40 

60* 

High 

Low 

Hiqh 

Low 

High 

Low 

Hinh 

Low 

1  vs.  2 

Wide 

31 

25 

27 

6 

14 

0 

22 

4 

Narrow 

0 

0 

0 

0 

0 

0 

0 

0 

1  vs.  3 

Hi  de 

32 

31 

40 

9 

34 

4 

32 

10 

Na  rrow 

0 

0 

0 

0 

0 

0 

0 

0 

1  vs.  4 

Wide 

39 

32 

38 

28 

40 

38 

32 

24 

Narrow 

8 

0 

0 

0 

0 

0 

0 

0 

1  vs.  5 

Wide 

39 

37 

39 

26 

40 

32 

34 

26 

Narrow 

3 

0 

0 

0 

9 

0 

0 

0 

♦Error  frequencies  doubled. 


a  smooth  reduction  in  size  from  the  second  to  subsequent  Eigenvalues  in  the 
unidimensional  case  when  tetrachoric  correlations  can  be  reliably  determined. 

As  item  difficulty  distributions  become  more  widespread,  however,  the  sampling 
errors  of  tetrachorics  also  become  more  variable.  Values  associated  with  very 
easy  and  very  difficult  items  are  huge. 

In  addition  to  the  substantial  sensitivity  to  the  distribution  of  item 
difficulties,  this  index  shows  increasing  overlap  with  small  N,  small  n, 
large  factor  intercorrelations ,  and  large  number  of  factors.  With  60  items 
and  the  narrow  range  of  item  difficulties,  overlap  is  zero  even  with  five  factors 
and  high  factor  correlations. 

Local  Independence  Index.  Summary  information  appears  in  Table  5. 

This  index  is  affected  similarly  by  the  five  parameters,  but  at  generally 
lower  amounts  of  overlap,  than  the  ratio  of  differences  index.  The  local 
independence  index  is  especially  effective  for  large  N^,  large  r^,  narrow  range 
of  difficulties,  and  for  the  distinction  between  one  and  two  factors. 

Pattern  of  Second  Factor  Signs.  Table  6  contains  the  overlap  information 
for  this  index.  Overall,  errors  in  the  detection  of  multidimensionality  are 
more  frequent  for  this  index  than  for  the  ones  based  on  ratios  of  Eigenvalue 
differences  and  on  local  independence,  but  the  effect  of  one  parameter  is 
quite  different.  The  two  indices  named  were  less  effective  in  the  wide  than 
in  the  narrow  range  of  item  difficulties,  but  the  pattern  of  second  factor 
loadinqs  reverses  that  effect.  The  latter  index  is  at  least  as  effective  as 
the  others  in  the  wide  range  of  item  difficulties  and  presumably  would  become 
more  effective  in  a  still  wider  ranqe. 

Size  of  First  and  Second  Factor  Loadinqs.  The  information  in  Table  7 
indicates  that  this  index  is  only  partially  redundant  with  the  pattern  of 


Table  5 


Amount  of  Overlap  in  Each  Cell 
Based  on  the  Local  Independence  Index 

N=125 


20 

30 

40 

60* 

High 

Low 

High 

Low 

High 

Low 

High 

Low 

1  vs.  2 

Wide 

24 

11 

26 

16 

17 

3 

4 

0 

Narrow 

22 

11 

18 

0 

0 

0 

0 

0 

1  vs.  3 

Wide 

32 

29 

22 

10 

18 

17 

28 

0 

Na  rrow 

25 

18 

22 

11 

7 

6 

0 

0 

1  vs.  4 

Wide 

31 

21 

30 

25 

25 

7 

22 

10 

Narrow 

35 

13 

22 

4 

22 

0 

8 

8 

1  vs.  5 

Wide 

30 

25 

26 

31 

34 

17 

36 

16 

Narrow 

34 

18 

31 

24 

33 

19 

10 

8 

N=500 

20 

30 

40 

60* 

High 

Low 

High 

Low 

High 

Low 

High 

Low 

1  vs.  2 

Wide 

5 

0 

0 

0 

0 

0 

0 

0 

Narrow 

0 

0 

0 

0 

0 

0 

0 

0 

1  vs.  3 

Wi  de 

8 

2 

5 

0 

1 

0 

0 

0 

Narrow 

0 

0 

0 

0 

3 

0 

0 

0 

1  vs.  4 

Wide 

23 

13 

7 

0 

2 

1 

0 

0 

Narrow 

2 

0 

5 

0 

2 

10 

0 

0 

1  vs.  5 

Wide 

32 

10 

18 

5 

16 

2 

2 

0 

Narrow 

16 

0 

11 

0 

22 

1 

0 

0 

•Error  frequencies 

doubled. 

Table  6 

Amount  of  Overlap  in  Each  Cell 

Based  on  Pattern  of  Second  Factor  Loadings 

N=  125 

20  30  40  60* 


High 

Low 

High 

Low 

High 

Low 

High 

Low 

1  vs.  2 

Wide 

32 

18 

33 

26 

21 

3 

0 

4 

Narrow 

39 

38 

36 

36 

35 

35 

24 

24 

1  vs.  3 

Wide 

36 

31 

25 

16 

19 

6 

12 

0 

Narrow 

38 

40 

37 

33 

36 

36 

32 

24 

1  vs.  4 

Wi  de 

35 

20 

16 

17 

10 

15 

16 

6 

Narrow 

40 

39 

35 

33 

40 

35 

24 

26 

1  vs.  5 

Wide 

39 

22 

28 

19 

19 

5 

10 

6 

Narrow 

39 

38 

37 

39 

36 

35 

26 

24 

N=500 

20 

30 

40 

60* 

High 

Low 

High 

Low 

High 

Low 

High 

Low 

1  vs.  2 

Wide 

6 

6 

0 

0 

3 

3 

0 

0 

Narrow 

31 

32 

27 

27 

4 

6 

0 

0 

1  vs.  3 

Wide 

11 

8 

0 

0 

15 

0 

0 

0 

Narrow 

34 

32 

27 

32 

7 

5 

0 

0 

1  vs.  4 

Wide 

27 

0 

8 

0 

23 

2 

4 

0 

Narrow 

25 

25 

36 

27 

0 

3 

0 

0 

1  vs.  5 

Wide 

8 

0 

19 

0 

19 

0 

6 

0 

Narrow 

25 

29 

32 

27 

7 

2 

0 

0 

♦Error  frequencies  doubled. 


Table  7 


Amount  of  Overlap  on  Each  Cell 


Based  on 

Pattern  of  First  and 

Second  Factor  Loadings 

N=125 

20 

30 

40 

60* 

High 

Low 

High  Low 

High 

Low 

High 

Low 

1  vs.  2 

Wide 

38 

23 

30  23 

13 

0 

0 

0 

Na  rrow 

24 

26 

25  26 

22 

16 

6 

8 

1  vs.  3 

Wide 

40 

29 

27  24 

14 

0 

4 

0 

Narrow 

36 

37 

30  27 

32 

26 

22 

12 

1  vs.  4 

Wide 

39 

36 

16  21 

14 

15 

6 

0 

Narrow 

40 

35 

35  32 

33 

27 

16 

14 

1  vs.  5 

Wide 

40 

34 

30  14 

15 

5 

6 

0 

Narrow 

40 

38 

36  37 

32 

34 

28 

22 

N=500 

20 

30 

40 

60* 

High 

Low 

High  Low 

High 

Low 

High 

Low 

1  vs.  2 

Wide 

6 

0 

0  0 

2 

0 

0 

0 

Narrow 

11 

3 

'  2  0 

0 

0 

0 

0 

1  vs.  3 

Wide 

24 

19 

0  0 

9 

0 

0 

0 

Na  rrow 

15 

14 

2  1 

3 

0 

0 

0 

1  vs.  4 

Wide 

23 

3 

20  0 

20 

0 

14 

0 

Narrow 

11 

13 

9  0 

2 

2 

0 

0 

1  vs.  5 

Wide 

18 

4 

25  3 

24 

0 

22 

0 

Narrow 

18 

16 

7  4 

2 

0 

0 

0 

*Error  freouencies 

doubled. 

ffWrfWWliWWWWWra.'.  P W  M  *,U  IT  i a  «i/i  v^i.  u>i  u  ■.»  *  ■  ■.: 


24 

second  factor  signs.  Overlap  for  the  fifth  index  also  tends  to  be  greater 
in  the  narrow  distribution  of  difficulties  than  in  the  wide,  but  the  effect 
is  smaller  than  for  the  fourth.  For  other  combinations  of  parameters  there 
is  no  substantial  advantage  for  one  or  the  other.  In  those  cases  where  #5 
does  appear  to  be  superior  to  #4,  it  tends  to  be  inferior  to  #3.  Overall, 
it  is  intermediate  in  its  characteristics  to  the  third  and  fourth  indices 
and  seems  to  have  no  unique  advantage. 

Main  Effects  of  the  Five  Indices.  Because  we  have  a  complete  factorial 
design,  it  is  possible  to  summarize  the  data  for  each  index  for  each  parameter 
by  adding  across  all  other  parameters  as  in  an  analysis  of  variance.  This 
will  reveal  more  clearly  the  comparative  strengths  and  weaknesses  of  the  five 
indices.  This  information  appears  in  Table  8. 

Indices  can  first  be  compared  by  notinq  the  sheer  amount  of  overlap  for 
each  of  the  15  main  effects.  Neglecting  the  fact  that  we  do  not,  at  the 
moment,  have  an  error  term,  it  is  seen  that  the  standardized  difference  in 
the  size  of  the  first  two  Eigenvalues  does  not  discriminate  best  for  any  level 
of  any  parameter  and  has  the  largest  total  amount  of  overlap.  The  local 
independence  index  has  the  lowest  total  amount  of  overlap  and  is  most  effec¬ 
tive  for  all  n_,  m,  N,  and  both  levels  of  factor  correlations.  It  is  not  at 
the  top,  however,  for  either  the  wide  or  narrow  distributions  of  item 
difficulties.  For  the  narrow  distribution  the  ratio  of  differences  in 
Eigenvalues  is  most  effective  and  for  the  wide  distribution  the  pattern  of 
second  factor  signs  is  at  the  top.  The  latter  index,  however,  is  second  only 
to  the  standardized  difference  in  total  number  of  errors. 

It  is  also  useful  to  compare  proportions  of  errors  of  inference 
within  a  given  parameter  across  the  five  indices.  The  two  indices  based  on 
Eigenvalues  are  least  sensitive  to  the  increase  in  N,  the  ratio  of  differences 
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Table  8 

Main  Effects  of  Number  and  Proportion  of  Errors 
Within  Treatments  for  Each  Index 


Number  of  Items 

Standardized  Ratio  of  Local  Patterns  of  Loadings 

Difference  Differences  Independence  Factors  1  and  2  Factor  2 


20 

1025 

.289 

626 

.311 

490 

.373 

753 

.418 

843 

.363 

30 

901 

.254 

533 

.265 

369 

.285 

506 

.281 

728 

.313 

40 

897 

.253 

514 

.255 

285 

.220 

362 

.201 

485 

.209 

60 

724 

.204 

340 

.169 

152 

.117 

180 

.100 

268 

.115 

Number  o* 

Factors 

1  vs.  2 

771 

.217 

341 

.169 

157 

.121 

304 

.169 

549 

.236 

1  vs.  3 

863 

.243 

440 

.219 

264 

.204 

447 

.243 

592 

.255 

1  vs.  4 

931 

.262 

599 

.298 

348 

.269 

496 

.275 

587 

.253 

1  vs.  5 

982 

.277 

633 

.314 

527 

.407 

554 

.308 

596 

.256 

Samnle 

Size 

N=1 25 

1992 

.562 

1128 

.560 

1072 

.827 

1430 

.794 

1654 

.712 

N=500 

1555 

.438 

885 

.440 

224 

.173 

371 

.206 

670 

.288 

Difficulty  Distribution 

Wi  de 

2108 

.594 

1859 

.924 

795 

.613 

792 

.440 

733 

.315 

Narrow 

1439 

.466 

154 

.076 

501 

.387 

1009 

.560 

1591 

.685 

Factor  Correlations 

Hi  nh 

2012 

.567 

1205 

.599 

874 

.674 

1078 

.599 

1309 

.563 

Low 

1535 

.433 

808 

.401 

422 

.326 

723 

.401 

1015 

.437 

Overlao 

3547 

2013 

1296 

1801 

2324 
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is  most  sensitive  by  far  to  the  distribution  of  item  difficulties.  The  local 
independence  index  is  most  sensitive  to  an  increase  in  the  number  of  factors 
and  comes  close  to  losing  its  #1  position  for  five  factors.  The  pattern  of 
second  factor  signs  stands  out  from  the  rest  for  the  wide  distribution  of 
item  difficulties  and  is  relatively  insensitive  to  the  number  of  factors 
being  compared. 

All  indices  show  the  expected  sensitivity  to  sample  size,  albeit  somewhat 
differentially,  to  obliquity  of  the  factors,  and  to  the  number  of  items.  An 
investigator  can  control  ^  and  usually  n  as  well  without  increasing  the  number 
of  major  factors  in  the  items.  Given  the  low  level  of  item  correlations  found 
typically  in  cognitive  tests,  large  samples  and  a  very  large  ratio  of  number 
of  variables  to  number  of  factors  by  ordinary  factor  analytic  standards  are 
important  design  considerations.  The  level  of  factor  correlations,  on  the 
other  hand,  cannot  be  controlled  by  the  investigator  except  indirectly  and 
inversely  with  respect  to  the  desired  direction.  As  mentioned  earlier,  if 
a  test  is  not  unidimensional  in  spite  of  the  item  writer's  best  efforts, 
the  multiple  factors  will  probably  be  highly  intercorrelated. 

Discussion 

This  research  has  documented  the  complexity  of  the  problem  of  dimen¬ 
sionality  in  binary  items.  No  one  index  of  the  five  we  have  tried  works 
best  in  all  combinations  of  our  parameters.  Only  one  index  of  the  five  we 
have  tried  can  be  rejected  in  toto:  namely,  the  difference  in  size  of 
Eigenvalues  of  the  first  two  principal  factors. 

Development  of  a  corpus  of  Monte  Carlo  runs  allowing  choice  of  an  index 
of  dimensionality  that  is  contingent  on  knowable  parameters  will  be  difficult. 
The  important  parameters  of  factor  intercorrelations  and  item  difficulties 
were  set  at  only  two  levels  each,  yet  each  varies  continuously.  Distributions 


of  item  difficulties  vary  alonq  more  than  one  dimension  as  well,  makinq 
generalization  from  several  fixed  distributions  hazardous.  A  priori 
difficulty  distributions  can  be  approximated  by  skillful  item  writers,  but 
factor  intercorrelations  cannot  be  kept  at  low  levels  by  skilled  test 
construction  practices.  Positive  correlations  among  factors  are  intrinsic 
to  the  cognitive  domain  and  are  not  trivial  in  size  between  group  factors 
that  are  widely  recoqnized  as  being  truly  different. 

In  spite  of  the  difficulties  we  believe  that  useful  recommendations  can 
be  developed.  In  a  preliminary  way  we  are  offering  to  readers  in  Appendix  A 
means  and  standard  deviations  of  the  three  nontraditional  methods  used  in  the 
current  research  for  each  cell  in  the  factorial  design.  Readers  have  a  feel 
for  the  Eigenvalues  of  tetrachoric  matrices,  but  lack  this  feel  for  the  new 
indices.  Readers  should  note  that  the  skewness  of  the  distributions  limits 
the  value  of  the  standard  deviations. 

Because  the  new  indices  are  quick  and  inexpensive  makes  further  work  on 
them  attractive.  On  the  basis  of  present  information,  full  information  factor 
analysis  of  Bock,  Gibbons,  and  Muracki  (1985)  solves  the  problem  of 
dimensionality  in  binary  items,  but  it  does  so  at  a  substantial  cost. 

Computer  time  for  this  method  increases  rapidly  as  the  number  of  items 
increases. 

For  the  present  we  are  working  on  several  possible  modifications  of  the 
indices  that  are  dependent  on  raw  score  and  standardized  variance-covariance 
matrices.  We  have  established  that  the  attractive  statistical  properties  of 
such  matrices  outweigh  their  dependency  in  binary  items  of  percent  passing. 
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Table  Al(cont.) 

N=125 

Four  Factors 

Wide  Narrow 

High  Low  High  Low 


i  n 

X 

Sx 

X 

Sx 

X 

Sx 

X 

Sx 

1  20 

.52 

.06 

.45 

.08 

.55 

.07 

.42 

.07 

1  30 

.54 

.05 

.49 

.07 

.54 

.07 

.47 

.07 

1  40 

.56 

.06 

.50 

.06 

.56 

.04 

.49 

.06 

1  60 

.62 

.03 

.56 

.05 

.58 

.07 

.52 

.08 

I 

Five 

Factors 

1  20 

.50 

.05 

.44 

.09 

.55 

.07 

.44 

.07 

1  30 

.54 

.05 

.51 

.09 

.57 

.06 

.51 

.08 

!  40 

.60 

.05 

.53 

.07 

.59 

.05 

.53 

.07 

J  60 

.65 

.02 

.56 

.07 

.62 

.04 

.56 

.05 

N=500 

One  Factor 

Wide  Narrow 

High  Low  High  Low 


n 

X 

Sx 

X 

Sx 

X 

Sx 

X 

Sx 

20 

.70 

.04 

.70 

.04 

.71 

.03 

.71 

.03 

30 

.71 

.03 

.71 

.03 

.71 

.02 

.71 

.02 

40 

.72 

.02 

.72 

.02 

.72 

.03 

.72 

.03 

60 

.75 

.02 

.75 

.02 

.75 

.01 

.75 

.01 

table  continues 
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Table  A1  (cont.) 

N=500 

Two  Factors 

Wide  Narrow 

High  Low  High  Low 


1 

X 

s* 

X 

Sx 

X 

sx 

X 

Sx 

o 

C\1 

.45 

.10 

.33 

.10 

.29 

.10 

.18 

.06 

30 

.36 

.07 

.28 

.06 

.24 

.07 

.14 

.06 

40 

.33 

.07 

.26 

.08 

.23 

.08 

.13 

.04 

60 

.32 

.08 

.20 

.06 

.21 

.06 

.13 

.04 

i 

if 

Three  Factors 

20 

.54 

.06 

.45 

.07 

.47 

.09 

.41 

.09 

\  30 

.52 

.07 

.44 

.10 

.43 

.09 

.36 

.08 

\  40 

.51 

.07 

.39 

.07 

.43 

.11 

.36 

.10 

60 

.50 

.07 

.43 

.05 

.46 

.07 

.40 

.09 

*< 

i 

Four 

Factors 

20 

.60 

.05 

.55 

.08 

.52 

.08 

.46 

.07 

30 

* 

.57 

.06 

.48 

.07 

.54 

.08 

.48 

.08 

40 

.58 

.04 

.51 

.06 

.52 

.07 

.44 

.09 

60 

.58 

.04 

.48 

.04 

.51 

.06 

.47 

.06 

1 

Five 

Factors 

K  20 

.61 

.06 

.57 

.05 

.59 

.07 

.54 

.06 

I  30 

.59 

.06 

.52 

.06 

.60 

.05 

.54 

.06 

1  40 

.63 

.04 

.55 

.06 

.59 

.05 

.51 

.05 

1  60 

.63 

.05 

.56 

.06 

.58 

.06 

.52 

.06 

I 

. 

•> ‘j,  *  ’jiV 

;*•  w> 

•* 

•.'v  vv^»v> 

WTO 

Table  A2 


Means  and  Variances  of  the  Index  Based  on  the  Pattern  of  Signs 
of  the  Second  Principal  Factor 

M=1 25 

One  Factor 

Wide  Narrow 


High  Low  High  Low 


n 

X 

Sx 

X 

Sx 

X 

Sx 

X 

Sx 

20 

17 

11 

17 

11 

36 

12 

36 

12 

30 

37 

21 

37 

21 

84 

33 

84 

33 

40 

43 

21 

43 

21 

147 

37 

147 

37 

60 

70 

57 

70 

57 

270 

85 

270 

85 

Two 

Factors 

20 

30 

14 

41 

7 

39 

7 

39 

7 

30 

68 

29 

88 

24 

100 

14 

96 

14 

40 

133 

46 

154 

25 

174 

26 

170 

28 

60 

360 

66 

357 

69 

400 

41 

362 

61 

Three 

Factors 

20 

34 

10 

36 

9 

40 

8 

36 

11 

30 

76 

29 

89 

15 

87 

20 

95 

11 

40 

144 

48 

144 

30 

172 

29 

166 

30 

60 

287 

87 

380 

42 

350 

74 

375 

55 

table  continues 


Table  A2  (cont.) 


N=125 

Four  Factors 

Wi de  Na  rrow 

High  Low  High  Low 


n 

X 

Sx 

X 

Sx 

X 

Sx 

X 

Sx 

20 

34 

12 

40 

11 

39 

10 

42 

9 

30 

90 

18 

86 

15 

91 

13 

98 

12 

40 

128 

35 

164 

38 

155 

26 

169 

21 

60 

211 

66 

316 

75 

400 

35 

394 

34 

Five 

Factors 

20 

30 

10 

38 

11 

38 

11 

39 

8 

30 

73 

31 

86 

19 

92 

18 

96 

20 

40 

139 

42 

143 

34 

171 

29 

170 

29 

60 

267 

109 

316 

103 

371 

49 

403 

47 

N=500 

One 

Factor 

Wi  de 

Narrow 

High 

Low 

High 

Low 

n 

X 

Sx 

X 

Sx 

X 

Sx 

X 

Sx 

20 

6 

4 

6 

4 

23 

12 

23 

12 

30 

9 

6 

9 

6 

46 

29 

46 

29 

40 

14 

16 

14 

16 

51 

28 

51 

28 

60 

28 

18 

28 

18 

74 

37 

74 

37 

table  continues 


Table  A2  (cont.) 


N=500 

Two  Factors 


Narrow 


X 

Sx 

X 

Sx 

X 

Sx 

X 

Sx 

33 

13 

37 

11 

40 

9 

42 

9 

88 

20 

88 

22 

98 

15 

95 

16 

143 

44 

152 

39 

164 

27 

165 

30 

344 

90 

375 

45 

392 

43 

396 

38 

Three  Factors 

25 

13 

30 

12 

39 

10 

41 

13 

74 

28 

90 

21 

101 

15 

92 

19 

105 

58 

157 

31 

156 

29 

171 

29 

313 

120 

367 

39 

384 

47 

389 

43 

Four  Factors 

25 

16 

35 

11 

40 

8 

41 

7 

50 

28 

82 

22 

88 

21 

93 

15 

101 

50 

144 

37 

163 

19 

172 

27 

217 

109 

342 

44 

405 

29 

351 

48 

Five 

Factors 

31 

13 

40 

8 

39 

7 

38 

7 

48 

35 

76 

23 

93 

18 

94 

15 

78 

50 

155 

25 

154 

28 

177 

26 

124 

91 

333 

92 

390 

43 

377 

49 
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Table  A3 

Means  and  Variances  of  the  Index  Based  on  the 

Products  of  the  First  and  Second  Principal  Factor  Loadings 

N=125 

One  Factor 

Wide  Narrow 

High  Low  High  Low 


n 

X 

Sx 

X 

Sx 

X 

Sx 

X 

Sx 

20 

.24 

.16 

.24 

.16 

.60 

.18 

.60 

.18 

30 

.32 

.15 

.32 

.15 

.86 

.38 

.86 

.38 

40 

.27 

.12 

.27 

.12 

1.07 

.35 

1.07 

.44 

60 

.38 

.13 

.38 

.13 

1.38 

.58 

1.38 

.58 

Two 

Factors 

20 

.47 

.24 

.71 

.23 

.84 

.21 

.86 

.31 

30 

.68 

.36 

.98 

.35 

1.32 

.20 

1.32 

.34 

40 

1.15 

.50 

1.23 

.25 

1.77 

.36 

1.88 

.44 

60 

1.87 

.46 

2.26 

.38 

2.98 

.49 

2.92 

.60 

Three  Factors 

20 

.42 

.19 

.53 

.18 

.78 

.21 

.75 

.29 

30 

.70 

.37 

.84 

.29 

1.09 

.32 

1.27 

.32 

40 

1.08 

.39 

1.16 

.24 

1.45 

.32 

1.61 

.39 

60 

1.30 

.48 

1.96 

.47 

2.21 

.53 

2.58 

.40 

table  continues 


Table  A3  (cont.) 


N=1 25 


Four  Factors 

Wide  Narrow 


Hiqh 

Low 

Hiqh 

Low 

n 

X 

Sx 

X 

Sx 

X 

Sx 

X 

S 

x 

20 

.47 

.23 

.61 

.16 

.66 

.23 

.76 

.22 

30 

.77 

.26 

.77 

.24 

1.06 

.28 

1.12 

.19 

40 

.88 

.36 

.98 

.36 

1.42 

.29 

1.54 

.33 

60 

.90 

.44  1 

.45 

.36 

2.42 

.32 

2.47 

.32 

Five 

Factors 

20 

.36 

.19 

.52 

.20 

.56 

.19 

.70 

.19 

30 

.62 

.30 

.75 

.22 

1.01 

.28 

1.10 

.27 

40 

.87 

.37 

.94 

.30 

1.41 

.28 

1.30 

.29 

60 

1.14 

.42  1 

.43 

.50 

1.96 

.35 

2.29 

.29 

N=500 

One 

Factor 

Wide 

Narrow 

Hinh 

Low 

Hiqh 

Low 

n 

X 

Sx 

X 

S 

X 

X 

Sx 

X 

Sx 

20 

.07 

.04 

.07 

.04 

.24 

.15 

.24 

.15 

30 

.08 

.05 

.08 

.05 

.25 

.17 

.25 

.17 

40 

.10 

.06 

.10 

.06 

.24 

.16 

.24 

.16 

60 

.16 

.08 

.16 

.08 

.29 

.10 

.29 

.10 

table  continues 


Table  A3  (cont. ) 


1 

N=500 

Two  Factors 

1 

i  “ 

Wide 

Narrow 

• 

j . 

Hi  qh 

Low 

High 

Low 

! 

4 

X 

Sx 

X 

S 

X 

X 

Sx 

X 

Sx 

>  20 

.46 

.22 

.57 

.20 

.78 

.28 

.88 

.18 

2  30 

.82 

.26 

.91 

.29 

1.29 

.29 

1.40 

.19 

j  40 

1.04 

.45 

1.22 

.38 

1.61 

.25 

1.79 

.27 

P  60 

1.70 

.48 

2.33 

.34 

2.76 

.42 

3.02 

.40 

Three  Factors 

1  20 

.32 

o 

CM 

• 

.47 

.22 

.62 

.23 

.68 

.18 

f  30 

.58 

.24 

.75 

CVI 

• 

1.10 

.17 

1.07 

.26 

40 

.67 

.42 

1.14 

.24 

1.27 

.35 

1.55 

.29 

1  60 

1.36 

.50 

1.80 

.22 

2.01 

.35 

2.10 

.36 

Four  Factors 

|  20 

.27 

.19 

.42 

.16 

.64 

.16 

.60 

.19 

1  30 

5 

.37 

.24 

.70 

.22 

.86 

.25 

1.13 

.19 

J  40 

4 

.61 

.40 

.94 

.32 

1.30 

.24 

1.38 

.20 

J  60 

.70 

.41 

1.42 

.33 

1.84 

.30 

2.19 

.25 

1 

Five  Factors 

I  20 

.42 

.17 

.46 

.13 

.53 

.14 

.58 

.16 

1  30 

.28 

.25 

.61 

*3* 

CNJ 

• 

.84 

.18 

.94 

.19 

I  40 

.39 

.28 

.89 

.27 

1.06 

.18 

1.28 

.19 

E  60 

.48 

.46 

1.40 

.44 

1.73 

.25 

2.00 

.33 
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